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PERSONAL  ELECTRONIC  AID  FOR  MAINTENANCE:  FINAL  SUMMARY  REPORT 


EXECUTIVE  SUMMARY _  j 

i 


Requirement: 

Printed  texts  provide  the  Information  necessary  to  support  the  operation 
and  maintenance  of  virtually  all  systems  and  equipment  in  the  Armed  Forces.  A 
steady  growth  in  the  size  of  technical  documentation  has  created  problems  in 
accessing  technical  information,  where  technicians  must  repeatedly  cross- 
reference  through  single  volumes  or  search  through  multiple  volumes  of  manuals 
to  obtain  needed  information.  Consequently,  critical  troubleshooting  proce¬ 
dures  are  bypassed,  and  the  likelihood  for  removal  of  nonfaulty  components 
increases.  This  problem  has  been  recognized  within  the  Department  of  Defense 
by  the  Computer-Aided  Acquisition  and  Logistics  Support  (CALS)  initiative,  and 
within  the  Army  by  a  draft  Required  Operational  Capability  for  the  Militarized 
Electronic  Information  Delivery  System  (MEIDS). 


Procedure: 

A  prototype  electronic  information  delivery  system  called  the  Personal 
Electronic  Aid  for  Maintenance  (PEAM)  was  developed  by  the  Joint  Services 
Manpower  and  Training  Technology  Development  program  for  evaluation  by  the 
Army  and  Naw .  The  Army  evaluated  PEAM  on  maintenance  tasks  for  the  Ml  Tank 
Turret;  the  V  *y  evaluated  it  on  maintenance  tasks  for  the  NATO  SEASPARROV 
Missile  System.  Because  of  technical  problems  vlth  the  four  prototypes, 
simulations  of  PEAM  on  commercially  available  equipment  were  also  used  in 
the  evaluation.  Comparisons  were  made  of  the  effectiveness  of  PEAM  and  its 
simulations,  relative  to  the  effectiveness  of  paper-based  technical  documen¬ 
tation,  in  terms  of  average  errors  per  task  and  time  to  task  completion. 


Findings: 

The  results  shoved  that  the  PEAM  approach  reduced  troubleshooting  errors 
by  about  a  6:1  ratio  in  the  Army  test,  and  about  a  5:1  ratio  in  the  Navy  test. 
In  the  nontroubleshooting  tasks  (Army  only),  there  was  about  a  2:1  advantage 
for  PEAM.  Time  to  task  completion  was  reduced  by  25X  in  the  Navy  test;  in  the 
Army  test,  the  time  to  task  completion  increased  slightly  because  of  the  slow 
display  rate  for  graphics.  Although  these  differences  are  sizable,  they 
should  be  interpreted  vlth  caution,  since  there  were  comparability  problems 
between  experimental  conditions.  These  results  demonstrate  the  potential  of 
electronic  delivery  for  reducing  maintenance  errors. 


Utilization  of  Findings: 

These  findings  contribute  to  the  formulation  of  future  electronic  techni¬ 
cal  information  delivery  systems  under  consideration  by  the  CALS,  MEIDS,  and 


vii 


other  programs.  Important  lessons  ere  learned  on  the  advantage  of 
dardized  authoring  and  operating  system  before  database  development 
advisability  of  pursuing  nondevelopmental  items  where  practicable. 
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PERSONAL  ELECTRONIC  AID  FOR  MAINTENANCE 
Final  Summary  Report 

INTRODUCTION 


General 

The  purpose  of  this  report  is  to  summarize  the  findings  of  the  Personal 
Electronic  Aid  for  Maintenance  (PEAM)  project,  which  was  completed  in  December 
1987.  The  findings  take  several  forms:  first,  there  are  the  empirical  results  from  user 
tests  in  the  Army  and  Navy;  second,  there  are  the  results  of  a  user  survey  regarding 
the  use  of  electronic  delivery  devices  instead  of  printed  technical  information;  and 
third,  there  are  the  lessons  learned  on  the  design  of  the  hardware  and  the 
development  of  the  databases.  Taken  together,  these  findings  form  the  basis  for 
proposed  enhancements  of  future  electronic  technical  information  delivery  systems. 

The  report  is  summary  in  nature.  More  detailed  information  on  user  tests  by  the 
Army  and  Navy  are  available  in  Schurman  and  Kincaid  (1988)  and  Smillie,  Nugent, 
and  Sander  (1988)  respectively. 


BasKflrjPMPtf 

Printed  texts — particularly  technical  manuals  (TMs) — provide  the  information 
necessary  to  support  the  operation  and  maintenance  of  virtually  all  systems  and 
equipment  in  the  Armed  Forces.  Nevertheless,  there  are  at  least  five  major  limitations 
inherent  in  paper-based  TMs  that  have  become  increasingly  evident  over  time: 

(1 )  The  sheer  volume  and  weight  of  paper  required  to  document  the 
necessary  technical  information; 

(2)  Problems  of  organizing  and  indexing  the  materia!  so  that  needed 
information  can  be  accessed  readily  by  users; 

(3)  The  relative  inflexibility  of  conventional  TMs  for  accommodating  the 
information  needs  of  users  with  varying  skill  levels; 

(4)  The  lack  of  a  dynamic  interactive  capability  between  the  TM  and  user; 

(5)  The  time  and  dollar  costs  associated  with  the  production,  distribution, 
updating,  and  correction  of  TMs. 

One  conseqi  jnce  of  the  steady  proliferation  of  military  technical  documentation 
involves  access  problems:  technicians  often  must  search  through  multiple  volumes  of 
TMs  to  obtain  needed  information.  A  dramatic  example  of  this  access  problem  is 
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described  in  a  report  issued  by  the  General  Accounting  Office  (1 979).  The  report 
notes  that,  to  isolate  and  repair  one  particular  radar  malfunction  on  a  C-141  aircraft, 
technicians  had  to  refer  to  and  cross-reference  165  pages  in  8  separate  documents. 

Another  example  of  the  cross-referencing  demands  of  a  TM  appears  in  a  three- 
page  troubleshooting  procedure  examined  during  the  evaluation  of  PEAM.  The  three- 
page  procedure  of  eleven  steps  makes  references  to  eight  other  procedures.  These 
references,  in  turn,  call  for  the  use  of  other  references.  A  total  of  38  references  are 
used  in  performing  the  basic  eleven-step  procedure.  This  total  represents  some 
duplication,  as  some  general  information  and  common  procedures  are  referenced 
from  several  procedures.  Including  these  procedures,  the  actual  number  of  referrals  is 
59. 


An  unfortunate  by-product  of  difficulties  in  accessing  required  technical 
information  is  that  it  tends  to  discourage  users  from  following  maintenance  procedures 
specified  in  the  TM.  Thus,  whenever  critical  troubleshooting  procedures  are  bypassed 
or  trial-and  error  practices  are  used  in  lieu  of  "doing  it  by  the  book,"  the  probability 
increases  for  higher  removal  rates  of  non-faulty  components.  Confirmatory  evidence 
of  this  assertion  is  provided  in  a  study  that  examined  the  causes  of  unnecessary 
removals  of  line  replaceable  units  on  Air  Force  avionic  equipment  (Rue  &  Lorenz, 
1983).  Results  of  this  study  showed  that  13  percent  of  1 ,008  unnecessary 
replacements  were  attributed  to  ineffective  technical  orders  (e.g.,  troubleshooting 
procedures  were  difficult  to  follow,  or  took  too  long  to  perform,  or  were  inaccurate). 
Deficiencies  in  TMs  were  also  reported  as  a  contributing  factor  to  unnecessary 
replacements  of  non-faulty  components  in  a  tri-service  field  study  by  Orlansky  and 
String  (1981).  Access  difficulties  can  lead  to  an  increase  in  maintenance  errors. 

Another  major  shortcoming  of  paper-based  TMs  is  that  the  information  they 
contain  is  static;  hence  they  cannot  readily  be  adapted  to  the  needs  of  the  user.  For 
example,  an  inexperienced  technician,  or  one  who  has  not  worked  on  a  particular 
equipment  for  some  time,  may  require  more  detailed  information  than  is  provided  in 
the  TM  to  perform  such  functions  as  locating  test  points,  removing,  replacing,  or 
adjusting  components.  Conversely,  highly  experienced  technicians  may  have 
performed  a  particular  maintenance  task  so  often  that  they  need  little  more  than  a 
checklist  to  ensure  that  no  maintenance  step  is  overlooked. 

One  attempt  to  solve  this  problem — particularly  for  entry-level  military 
maintenance  technicians — is  the  development  of  highly  proceduralized  and  illustrated 
manuals,  known  as  job  performance  aids  (JPAs).  Numerous  studies  have  shown 
JPAs  to  bo  effective  both  in  terms  of  their  method  of  reorganizing  and  reformatting 
technical  information  and  in  improving  job  task  performance.  (See  Foley  &  Camm, 
1972;  Johnson,  Tnomas  &  Martin,  1977;  and  Smillie,  1985  for  reviews  of  the  JPA 
literature.)  These  advantages  notwithstanding,  several  problems  have  inhibited  the 
use  of  JPAs  in  military  settings.  For  example,  because  JPAs  are  intended  primarily  as 
supplements  to,  rather  than  as  replacements  for  conventional  TMs,  they  add  to  the 
proliferation  of  technical  information.  Another  problem  is  the  time  factor  since  the 
development  of  JPAs  frequently  requires  substantial  investment  both  in  terms  of  front- 
end  analyses  and  subsequent  verification  of  the  information.  These  factors  result  in 
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JPAs  that  are  equally  as,  and  in  some  cases,  more  costly  to  design,  produce,  and 
update  than  conventional  TMs. 

Paper-based  approaches  to  technical  information  do  not  offer  a  dynamic, 
interactive  capability  between  the  information  source  and  user.  This  is  true,  of  course, 
for  both  TMs  and  JPAs.  They  cannot,  for  example,  provide  hands-free  access  to,  or 
alternative  methods  for,  delivery  of  needed  technical  information,  such  as  through 
text-to-speech  output  devices.  Neither  can  they  highlight  or  animate  selected  portions 
of  the  technical  information  "on  demand,"  such  as  emphasizing  the  physical  location  of 
components,  or  highlighting  the  path  of  a  signal.  The  application  of  new  technologies 
to  promote  the  advantages  of  JPAs  to  the  users  and  to  address  the  inherent  problems 
of  paper-based  approaches  were  motivating  factors  for  the  development  of  PE  AM. 

Although  electronic  iechnica!  information  delivery  systems  offer  advantages 
over  TMs  and  JPAs,  they  do  not  necessarily  avoid  the  time  and  dollar  costs  associated 
with  the  production,  distribution,  and  corrections  of  technical  information.  These 
problem  areas  might  be  mitigated  through  the  development  of  authoring  systems  and 
electronic  distribution  networks. 

During  the  development  and  evaluation  of  PEAM,  other  applications  of 
electronically  delivered  technical  information  were  under  investigation.  These  include: 

•  CMAS  -  Computer-based  Maintenance  Aid  System.  See  Clay  (1 986) 
and  Nugent,  Sander,  Johnson,  and  Smillie  (1987). 

•  NTIPS  -  Navy  Technical  Information  Presentation  System.  See  Fuller, 
Post  and  Mavor  (1987). 

•  EMPS  -  Electronic  Maintenance  Publication  System.  See  MRSA  (1987). 

The  breadth  of  these  projects  underscores  the  Department  of  Defense  (DoD) 
interest  in  the  topic  of  electronic  delivery  of  digital  logistic  technical  information. 

Indeed,  DoD  has  established  a  Computer-aided  Acquisition  and  Logistic  Support 
(CALS)  program  to  increase  capabilities  to  receive,  distribute,  and  use  logistic 
technical  information  in  digital  form.  CALS  is  also  oriented  toward  accelerating  the 
automation  of  contractor  processes  for  generating  logistic  technical  information 
products.  A  near-term  goal  of  CALS  is  to  develop  recommendations  for  functional 
specifications  relating  to  delivery  of  TM  information  to  users  in  electronic  form  for 
interactive  access.  The  results  of  the  PEAM  project  will  hopefuliy  contribute  to  this 
goal. 


PEAM  Development 

The  PEAM  project  began  with  a  concept  definition  effort  jointly  sponsored  by  the 
Naval  Training  Systems  Center  (NTSC),  at  that  time  the  Naval  Training  Equipment 
Center,  and  the  Army  Project  Manager  for  Training  Devices  (PM-TRADE).  The  initial 
effort,  which  was  completed  during  the  third  quarter  of  FY82,  resulted  in  an  overall 
concept  formulation  and  design  analysis.  Next,  detailed  sf  fications  were 
developed  for  designing  a  portable  maintenance  informatic.  ^elivery  system  and 
supporting  software.  During  the  same  year,  a  memorandum  of  understanding  was 
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issued  between  the  Army  Research  Institute  and  the  Naval  Training  Systems  Center. 
The  memorandum  established  agreements,  support,  and  resource  responsibilities  for 
the  pntotype  development  phase  of  PEAM.  The  Joint  Services  Manpower  and 
Training  Technology  Development  Program  sponsored  the  development  and 
evaluation  of  the  PEAM  prototype.  Major  milestones  of  the  project  were: 


FY80 

FY82 

FY82 

FY86 

FY87 


Formulated  the  conceptual  definition  for  PEAM. 
Completed  the  hardware  and  software  development 
specifications. 

Contracted  for  the  development  of  PEAM. 
Contractor  delivered  four  PEAM  prototypes. 
Completed  test  and  evaluation. 


Description  of  PEAM 

The  PEAM  software  includes  a  front-end  system  for  converting,  compiling,  and 
storing  maintenance  databases  onto  computer-readable  media,  termed  memory 
modules.  The  modules,  in  turn,  are  plugged  into  the  computer-based  delivery  system 
for  subsequent  use  by  the  field  technician.  Collectively,  these  subsystems  are 
expected  to  have  advantages  over  existing  paper-based  maintenance  documentation 
in  that  they:  (1 )  offer  greater  versatility  in  the  way  information  can  be  organized  and 
accessed;  (2)  reduce  search  time  for  the  technician  by  providing  only  the  necessary 
and  sufficient  information  to  perform  each  particular  maintenance  task;  and  (3)  use  a 
medium  that  can  be  readily  updated  and  easily  stored. 

With  respect  to  its  physical  configuration,  PEAM  is  a  portable  device 
(approximately  the  size  of  a  briefcase)  that  contains  the  following  major  components, 
displayed  in  Figure  1 : 

1 .  A  host  controller  or  "electronic  brain"  which  recognizes  human  speech;  controls 
the  presentation  of  user-requested  text,  graphics,  and  audio  information;  and 
performs  automatic  electronic  checks  of  internal  components  to  ensure  that  they 
are  working  properly. 

2.  Power  cords  that  link  the  main  unit  to  a  portable  battery  pack  or  conventional 
AC  power  sources,  allowing  switching  between  12V  DC,  28V  DC,  110V  AC,  and 
220V  AC. 

3.  A  headset  with  a  built-in  microphone  and  earphone  that  provides  voice- 
activated  cursor  control/interaction  with  the  display  screen  and  reception  of 
synthetic  voice  output,  respectively. 

4.  A  mass  memory  cartridge  that  can  store  approximately  1 75,000  words  and  300 
graphics.  This  equates  to  up  to  4,000  narrative  pages,  when  a  data  compression 
technique  is  applied. 

5.  A  detachable  interface  module  that  can  be  hand-held  or  easily  mounted.  The 
module,  which  is  connected  to  the  host  controller  by  a  1 2-foot  long  coaxial  cable, 
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contains  the  following  components:  (a)  an  electroluminescent  display  screen,  (b)  an 
eight-function  keypad,  and,  (c)  a  speaker  for  synthetic  voice  output. 


1  -  Host  controller  4  -  Memory  cartridge 

2  -  Power  cords  5  -  Interface  module 

3  -  Headset 


Figure  1. 

Major  Components  of  PEAM 
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The  PEAM  interface  module  can  present  information  either  visually  or  aurally.  These 
options  can  be  used  separately  or  in  combination.  The  visual  display  consists  of  a 
monochrome,  5"  x  7"  dot  matrix,  thin-film  electroluminescent  (TFEL)  panel  that  can 
display  up  to  350  x  490  lines  (171K  pixels).  Textual  information  presented  on  the 
TFEL  panel  can  be  "read"  to  the  user  by  means  of  a  text-to-speech  synthesizer.  In 
addition,  input  to  the  device  can  be  controlled  either  manually  or  by  voice  activation. 
The  manual  input  consists  of  eight  pushbutton  function  keys,  each  associated  with  a 
specific  operational  command  (i.e.,  MENU,  SELECT,  NEXT,  LAST,  BACKUP,  SPEAK, 
YES,  NO).  The  same  operational  commands  can  be  voice-activated  by  means  of 
speaking  into  the  built-in  headpiece  microphone.  To  activate  this  feature,  the 
technician  vocalizes  each  of  the  eight  function  commands  several  times,  thereby 
enrolling  these  commands  in  the  module's  memory.  When  the  technician  speaks  a 
command,  the  module  performs  the  requested  function  just  as  if  the  appropriate 
command  key  had  been  pressed. 

The  structure  of  the  PEAM  database  is  basically  hierarchical;  the  control 
starts  with  a  top  frame  containing  objects  that  link  in  a  fixed  manner  with  other 
frames.  These  frames,  in  turn,  contain  objects  that  link  with  other  frames.  A 
single  page  of  a  JPA,  for  instance,  might  be  contained  on  a  single  frame,  or  be 
linked  across  several  frames.  The  result  is  a  database  of  a  high  degree  of 
interconnectivity  with  singla  points  of  entry  and  multiple  paths  to  exit.  It  is  this 
early-exit  feature  that  allows  experienced  technicians  to  traverse  the  database 
rapidly,  while  accommodating  the  less  experienced  technicians'  requirements 
for  detailed  information  review.  This  feature  also  automates  the  cross- 
referencing  of  information. 

The  database  transferred  to  the  memory  module  utilized  a  data 
compression  scheme  that  allowed  for  the  compression  of  up  to  4,000  pages  of 
technical  documentation  into  one  megabyte  of  memory.  The  foundation  of  this 
compression  scheme  relies  on  the  fact  that  the  vocabulary  of  maintenance  is 
limited,  allowing  a  single  byte  to  address  a  look-up  table  of  words.  A  frame, 
then,  contains  pointers  to  its  constituent  words  which  are  assembled  on-the-fly 
when  a  single  frame  is  composed  for  display.  The  same  concept  is  applied  to 
common  phrases  and  sentences. 

Problems  In  Development 

The  development  of  PEAM  and  its  databases  was  accompanied  by  numerous 
setbacks  and  impediments,  as  is  sometimes  the  case  in  the  development  of  a  state-of- 
the-art  prototype.  There  was  an  interruption  in  funding,  which  caused  continuity 
problems  in  the  contracts  and  extended  the  test  and  evaluation  schedule.  There  were 
significant  engineering  challenges,  especially  in  the  connecting  tolerances  on  the 
display  panel,  and  on  the  design  of  the  power  supply.  The  satisfactory  completion  of 
the  databases  delayed  the  start  of  the  test  program,  and  ultimately  altered  the  course 
of  the  evaluations.  The  database  development  process  was  clearly  hampered  by  the 
absence  of  an  authoring  system.  Simulations  of  PEAM  on  commercially  available 
equipment  were  needed  to  complete  the  evaluation  of  the  PEAM  approach  to 
maintenance.  Nevertheless,  there  was  sufficient  progress  on  the  PEAM  prototypes 
and  databases  to  allow  a  useful  evaluation  to  proceed. 
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TEST  AND  EVALUATION 


Overview 

The  purpose  of  the  test  and  evaluation  was  to  compare  the  effectiveness  of 
PEAM  to  that  of  traditional,  paper-based  approaches  to  equipment  maintenance. 
Since  PEAM  is  a  prototype  electronic  technical  information  delivery  system,  the  test 
and  evaluation  was  directed  at  general  questions  regarding  its  advantages  and 
limitations  in  comparison  to  traditional  methods  of  using  maintenance  documentation. 
These  questions  concerned  accuracy  in  executing  a  maintenance  prodedure,  and 
time  to  complete  a  maintenance  procedure. 

Because  PEAM  was  a  joint  service  effort,  it  was  evaluated  in  both  an  Army  and 
a  Navy  environment.  In  general,  each  evaluation  sought  to  determine  the  relative 
effect  of  PEAM  on  maintenance  errors  and  time  to  repair.  The  Army  evaluation  used 
both  the  PEAM  hardware  and  a  commercially  available,  laptop-type  computer  with 
similar  display  features.  The  Army  evaluation  was  conducted  on  the  details  of 
maintenance  for  the  turret  of  the  Ml  tank.  The  Navy  evaluation  was  directed  at  a  more 
global  level  of  maintenance  performance  on  the  NATO  SEASPARROW  Surface 
Missile  System,  and  used,  in  effect,  a  simulation  of  PEAM  on  a  workstation-type 
computer. 

Each  evaluation  will  be  summarized  separately;  then,  subsequent  tables  in  this 
report  will  present  combined  results. 


Army  Test  and  Evaluation  Description 

The  technical  objective  was  to  test  PEAM  on  maintenance  tasks  in  a  school 
environment,  and  to  compare  it  to  technical  manuals  in  supporting  the  needs  of  Army 
mechanics.  The  particular  domain  chosen  for  the  test  was  tank  turret  maintenance  for 
the  Ml  Tank.  The  approach  involved  quantitative  data  collection  of  troubleshooting 
performance  and  qualitative  assessment  by  means  of  a  user  evaluation  questionnaire. 
The  Army  test:  (1 )  included  corrective  and  other  maintenance  tasks  in  addition  to 
troubleshooting  tasks;  (2)  used  the  actual  PEAM  device  as  well  as  another  portable 
microcomputer  (a  Grid  Compass)  which  emulated  the  PEAM;  and  (3)  was  conducted 
in  a  simulated  shop  environment  rather  than  a  controlled  laboratory  environment. 
Tasks  in  the  test  included  troubleshooting  (6  tasks),  adjust/align  (3  tasks), 
service/maintain  (2  tasks),  remove  (11  tasks),  and  install/replace  (12  tasks).  The  test 
and  evaluation  was  conducted  by  a  contractor  to  the  Army  Research  Institute. 


Method 

There  were  14  technicians  in  the  test,  5  in  the  control  group  using  technical 
manuals  to  perform  the  tasks  and  9  in  the  experimental  group  using  PEAM  or  its 
simulator  to  perform  the  tasks.  Subjects  were  qualified  mechanics  for  the  M60  tank 
turret,  but  not  for  the  Ml  tank  turret.  Each  subject  was  observed  performing  tasks  on  a 
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Ml  tank  turret  simulator  or  in  the  tank  itself  at  the  Armor  School,  Ft.  Knox,  Kentucky.  A 
qualified  instructor  from  the  school  was  always  available.  Faults  were  inserted  in  the 
simulator  or  the  tank  for  the  troubleshooting  tasks.  Technicians  performed  each  task 
using  the  tools  and  procedures  that  ordinarily  are  used  at  the  Armor  School. 

The  quantitative  data  were  errors  per  task  and  time  to  complete  each  task.  The 
error  data  included  errors  in  good  practice,  errors  in  orientation,  technique,  and 
specification,  recursive  errors,  and  dangerous  or  destructive  errors.  The  error  data 
were  summated  within  tasks  to  derive  a  measure  of  overall  average  errors  per  task. 


Test  and  Evaluation  Description 

The  technical  objective  was  to  test  PEAM  as  a  maintenance  aid  for  the  NATO 
SEASPARROW  Surface  Missile  System  (NSSMS).  Specifically,  this  effort  compared 
the  efficiency  and  effectiveness  of  an  electronically  delivered  technical  information 
system  to  that  of  a  paper-based  documentation  system  in  supporting  the  information 
needs  of  NSSMS  maintenance  technicians.  The  approach  involved  quantitative  data 
collection  of  troubleshooting  performance  and  qualitative  assessment  by  means  of  a 
user  evaluation  questionnaire.  The  test  and  evaluation  was  conducted  by  the  Navy 
Personnel  Research  and  Development  Center. 

PEAM  operational  software  was  used  to  construct  the  electronically  delivered 
NSSMS  technical  information  database.  Technical  difficulties  in  loading  the  database 
onto  the  PEAM  memory  modules,  however,  precluded  the  use  of  PEAM  hardware 
when  tests  were  scheduled  to  be  conducted.  As  a  result,  alternative  hardware,  a  Sun 
3/75  computer  without  a  voice  capability,  was  used  to  simulate  the  operation  of  PEAM. 
With  the  exception  of  minor  differences  in  the  method  of  accessing  information  (e.g., 
use  of  mouse  control  versus  pushbutton  function  keys),  the  organization  and  format  of 
the  database  were  identical  to  that  of  PEAM. 


Method 

There  were  28  participants  in  the  evaluation,  15  instructors  and  7  students  from 
the  NSSMS  C-school  at  the  Combat  Systems  Technical  School  Command  in  Mare 
Island,  California,  and  6  NSSMS  technicians  from  fleet  units  in  San  Diego,  California. 

In  the  performance  evaluation,  the  NSSMS  technicians  attempted  to  fault- 
isolate  two  problems,  one  using  the  PEAM  simulation  to  access  the  required  technical 
information,  and  the  other  using  the  current  paper  technical  manuals.  The  faults, 
which  were  simulated  on  a  separate  microcomputer,  were  representative  of  the  types 
of  NSSMS  malfunctions  encountered  in  the  fleet.  In  addition  to  troubleshooting,  the 
technicians  were  required  to  identify  specific  parts  of  information  and  to  locate 
narrative  information  concerning  the  function(s)  of  the  circuit  that  contained  the 
malfunction.  Quantitative  data  included: 

-  Success  criterion:  successfully  identifying  the  faulted  component 
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-  Time  spent  troubleshooting:  overall  time  spent  attempting  to  isolate  the  fault 

-  False  replacements:  conclusion  that  a  component  was  the  source  of  the  fault 
when  it  was  not 

-  Errors  of  commission:  checking  test  points  or  performing  tests  that  were 
unrelated  to  the  fault  symptoms 

-  Parts  information:  success  in  locating  and  time  to  locate  the  correct  parts 
information 

-  Functional  description:  success  in  locating  and  time  to  locate  the  narrative 
functional  description 


As  in  the  Army  test,  error  data  were  summated  within  tasks  to  derive  an 
overall  measure  of  average  errors  per  task. 


User  Questionnaire 

The  attitudes  of  test  participants  toward  the  use  of  PEAM  were  assessed  by 
means  of  a  user  evaluation  questionnaire.  All  14  Army  and  28  Navy  participants 
completed  the  questionnaire.  The  questionnaire  contained  items  relating  to:  (1)  the 
physical  features  of  the  electronic  delivery  device  (8  items);  (2)  the  operation  and 
software  features  (6  items);  (3)  ease  of  access  to  information  (3  items);  and  (4)  the 
effectiveness  of  the  information  (5  items).  Open-ended  comments  on  the  likes  and 
dislikes  of  PEAM  were  also  collected. 
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RESULTS  AND  DISCUSSION 


PEAM  Effectiveness 

Table  1  summarizes  the  scopes  of  the  Army  and  Navy  evaluations.  The  results 
of  these  demonstrated  the  potential  effectiveness  of  electronic  technical  information 
delivery  systems  over  conventional  paper-based  approaches  (Table  2).  These  results 
should  be  interpreted  with  caution,  however,  as  there  were  occasional  differences 
between  the  databases  of  the  technical  manuals  and  electronic  devices.  These 
differences  were  due  to  problems  in  the  compilation  process;  when  they  occurred,  the 
differences  were  small.  The  small  sample  size  in  the  Army  evaluation  leaves  room  for 
the  influence  of  individual  differences.  Still,  the  robustness  of  the  differences  between 
technical  manuals  and  electronic  technical  information  delivery  denotes  the  superiority 
of  electronic  delivery.  There  were  no  major  differences  between  PEAM  and  its 
simulations. 

The  advantage  of  PEAM,  in  terms  of  error  resolutions,  was  more  pronounced  for 
troubleshooting  tasks,  where  a  nearly  6:1  advantage  was  found,  versus  an  advantage 
of  only  2:1  for  a  non-troubleshooting  maintenance  task  (Table  3).  This  difference  is 
probably  the  result  of  an  increase  in  cross-referencing  and  the  adjunct  increase  in 
cognitive  load  of  technicians  sorting  through  multiple  pages  and  volumes  while  trying 
to  problem-solve  a  fault.  Apparently,  when  referencing  is  automated,  technicians  can 
better  focus  their  attention,  which  reduces  the  likelihood  of  error.  Furthermore,  a  more 
detailed  analysis  of  the  Navy  evaluation  indicated  an  advantage  of  electronic  delivery 
for  the  more  complicated  troubleshooting  problem,  in  terms  of  number  of  actions 
required  by  the  technicians.  In  the  second  troubleshooting  problem,  where  the 
number  of  actions  were  small,  the  use  of  technical  manuals  was  equivalent  to 
electronic  delivery  in  terms  of  errors  per  task. 

The  time  factor  in  tne  Army  evaluation  was  confounded  by  a  slow  display  time 
exceeding  15  seconds  for  graphics  on  the  PEAM  device.  However,  in  the  Navy 
evaluation  where  display  times  on  the  workstation  were  rapid,  there  was  nearly  a  25% 
reduction  in  task  completion  time  when  using  electronic  delivery  (Table  3). 
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Table  1 

Scope  of  Army  and  Navy  PEAM  Field  Tests 


Technical  Manuals 


Army 


M-1  Tank  Turret 

328  Task  Observations 
5  Subjects 
220  Hours 


Navy 

NATO  SEASPARROW 

56  Task  Observations 
28  Subjects 
50  Hours 


PEAM/Electrontc  Delivery 


347  Task  Observations 
9  Subjects 
290  Hours 


56  Task  Observations 
28  Subjects 
50  Hours 


Table  2 

Combined  Test  Results  for  all  Tasks 


Technical  -Manuals  PEAM/Eleclronic  Delivery 

Average  Errors/Task 

Army  1 .37  .48* 

Navy  5.70  .94* 

Army  test  showed  reduction  in  errors  by  nearly  3:i. 

Navy  test  showed  reduction  in  errors  by  nearly  6:1. 

Average  Time/Task 

Army  16.4  min.  18.9  min.t 

Navy  43.9  min.  33.1  min.* 


*  Statistically  significant  difference  (p<.05)  by  parametric  tests. 

t  Additional  time  for  PEAM  group  in  Army  test  due  to  long  time  (>1 5  sec.  per  frame)  for  graphics  to 
appear. 
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Table  3 

Analysis  of  army  and  Navy  Test  data 


Army  test  showed  reduction  in  troubleshooting  errors  by  nearly  5:1 . 
Navy  test  showed  reduction  in  troubleshooting  errors  by  nearly  6:1. 
Army  test  showed  reduction  in  non-troubleshooting  errors  by  nearly  2:1. 


Average  Time/Task 


Troubleshooting 

Army 

37.0  min. 

41.6  min.t 

Navy 

43.9  min. 

33.1  min.* 

Non-Troubleshooting 

16.1  min.t 

Army 

12.0  min. 

Note:  Navy  test  was  restricted  to  troubleshooting  tasks. 


*  Statistically  significant  difference  (p<.05)  by  parametric  tests. 

t  Additional  time  for  PEAM  group  in  Army  due  to  long  time  (>15  sec.  per  frame)  for 
graphics  to  appear. 
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The  voice  systems  were,  unfortunately,  not  given  a  substantive  test  due  to 
technical  problems.  The  Army  test  used  the  voice  systems  initially,  and  the  users 
acknowledged  the  advantages,  particularly  in  restricted  spaces.  The  intelligibility, 
however,  was  not  sufficiently  high  to  maintain  continual  use.  This  advance  in  "hands¬ 
free"  use  had  much  intuitive  appeal,  but  there  are  no  direct  data  from  the  PEAM 
evaluations  to  support  its  use.  A  related  project  (Nugent,  1988),  which  would  have 
tested  the  PEAM  voice  technology  had  it  been  available  in  the  Navy  test,  found 
advantages  in  audio  supplements  to  textual  displays. 

An  apparent  problem  of  the  users  "getting  lost"  in  the  database  was  noticed 
during  the  Army  evaluation.  There  was  no  facility,  such  as  an  on-line  map,  to  indicate 
to  the  user  where  he  was  in  the  procedure.  This  seemed  to  confuse  some  technicians. 

Besides  the  empirical  results,  there  was  a  collective  experience  gained  by 
those  involved  with  various  phases  of  the  project  that  offers  a  useful  perspective  on  the 
development  decisions  that  later  led  to  problems  during  the  test  phase.  Leading  the 
list  was  the  absence  of  an  authoring  system  and  format  standards  which  resulted  in 
labor-intensive  efforts  to  transfer  the  paper-based  information  to  digital  storage.  This  is 
in  sharp  contrast  to  the  current  paper-based  system,  which  is  standardized  to  a  degree 
through  such  documents  as  Military  Specification  MIL-M  3874. 

The  decision  in  1981  to  design  and  build  a  prototype  rather  than  to  acquire 
commercial  off-the-shelf  equipment  was  a  consequence  of  the  uncertainty  at  that  time 
of  the  course  of  near-term  advances  in  commercial  developments.  Given  the 
availability  of  commercial  equipment  to  simulate  PEAM  at  the  time  of  its  evaluation,  the 
decision  remains  debatable. 


Qugstlfljiitalrft  jteaulte 

Inii.al  analysis  of  the  user  evaluation  questionnaire  data  found  no  differences 
among  the  participants  either  in  terms  of  their  branch  of  service  (Army  or  Navy)  or  their 
level  of  maintenance  experience  with  either  system  (M-1  Abrams  Tank  for  the  Army; 
NATO  SEASPARROW  Surface  Missile  System  for  the  Navy).  Specifically,  a  multiple 
analysis  of  variance  was  used  to  analyze  the  mean  ratings  of  users'  responses  to  the 
four  categories  of  questions.  The  results  of  the  User  Evaluation  Questionnaire  are 
summarized  in  Table  4. 

Overall,  the  Navy  technicians  rated  the  electronic  presentation  system  from 
"unsatisfactory"  to  "highly  satisfactory."  Although  various  physical  characteristics  were 
assessed  in  the  first  category  of  questionnaire  items,  a  single  mean  (3.54)  reflects  the 
similarity  in  the  users'  ratings  of  those  items.  Similarly,  the  technicians  rated  the 
adequacy  of  the  computer  operation  and  software  features  in  the  "satisfactory"  to 
"highly  satisfactory"  range  (x=3.48).  Use  of  the  electronic  presentation  system  was 
also  perceived  by  the  Army  and  Navy  technicians  as  an  improvement  in  terms  of 
efficiency  and  effectiveness  over  conventional  technical  manuals  as  a  source  for 
maintenance  information. 
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A  review  of  the  open-ended  comments  found  that  the  technicians’  overall 
impression  of  the  electronic  delivery  system  was  generally  quite  favorable.  The 
primary  dislike  concerned  the  size  of  the  display  screen  for  presenting  schematic-type 
diagrams.  The  most  frequent  negative  comment  provided  in  this  regard  was  that  the 
diagrams  were  too  small,  thus  precluding  a  view  of  the  diagram  in  its  entirety. 
Additional  negative  comments  concerned  the  long  delays  encountered  when  the 
PEAM  software  "drew”  graphic  information  on  the  screen,  in  some  instances  15-90 
seconds. 


Table  4 

Summary  of  User  evaluation  Questionnaire 
for  Combined  Army/Navy  Sample 

Feature  X 

Physical  Features  3.54 

Operation  &  Software  Features  3.48 

Effectiveness  of  Information  3.62 

Ease  of  Accessing  Information  2.37 

Scale  Values: 

— Physical  Features:  1  =  Unsatisfactory  5  =  Outstanding 

—Operation  &  Software:  1  *  Unsatisfactory  5  =  Outstanding 

—Effectiveness:  1  =  Slightly  Less  _  5  *  Significantly  More 

— Accessing  Information:  1  =  Less  Difficult  5  *  More  Difficult 

Note:  Low  value  for  Accessing  Information  reflects  a  positive  response. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


Improvements  in  a  technician's  performance  on  a  maintenance  task  can 
potentially  be  increased  by  electronic  technical  information  delivery  systems. 
Technicians  are  favorably  disposed  to  their  use,  and  they  perceive  improvements  in 
effectiveness  over  conventional,  paper-based  approaches.  In  the  conclusions  and 
recommendations  listed  below,  PEAM  refers  both  to  the  prototype  and  simulations. 

Findings  and  Conclusions 

1)  Use  of  PEAM  reduced  procedural  maintenance  errors. 

2)  Use  of  PEAM  reduced  time  to  identify  corrections,  except  where  long  display 
times  were  due  to  c  lipment  used  in  the  test. 

3)  The  PEAM  approach  was  more  effective  as  problems  were  more  difficult  in 
terms  of  branching  and  cross-referencing, 

4)  Users  were  receptive  to  the  advantages  and  usability  of  electronic  delivery. 

5)  Menu-driven  control  of  a  hierarchical,  fixed-link  Job  Performance  Aid  (JPA) 
format  was  a  successful  method  of  procedural  representation. 

The  first  four  conclusions  are  based  on  quantitative  findings  from  the  PEAM 
tests.  The  fifth  conclusion  is  a  qualitative  judgment,  as  there  was  no  comparison 
between  various  electronic  representation  formats. 

Based  on  these  findings  and  conclusions,  and  on  the  knowledge  acquired 
during  the  course  of  the  project,  five  recommendations  are  proposed  as  follows. 


Recommendations 

1 )  Future  developments  of  electronic  delivery  of  technical  information  should 
take  into  account  the  functional  features  of  PEAM  enhanced  with  the  lessons 
learned  from  the  test  and  evaluation.  The  features  are  described  in 
Appendix  A. 

2)  Standards  for  an  operating  system,  database,  and  authoring  system  should 
be  established  prior  to  selection  or  development  of  delivery  hardware. 

3)  The  digitized  approach  to  database  development  should  be  pursued  in 
future  delivery  systems,  as  it  has  demonstrated  advantages  during 
development  and  economies  in  memory  requirements. 

4)  Voice  technology  as  an  input/output  medium  should  be  considered  further  in 
future  delivery  systems.  Its  advantages  were  recognized  by  users. 
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5)  Slow  display  response  times  negatively  affected  technician  performance. 
Response  times  should  be  contained  within  human  factors  standards  of 
about  two  seconds. 
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Appendix  A 


Enhanced  Functional  Characteristics 
Based  on  Lessons  Learned  from  PEAM  Test  and  Evaluation 


Functional 

Characteristic 

PEAM 

Evaluation  Findings 

Degree  of  Support 
from  Evaluation 

Screen  Display 

— Size 

5"x7"  marginally  acceptable  for  most 
applications.  Exception  is  complex 
graphics  such  as  signal  flow  diagrams 
which  need  larger  displays  (or  different 
formats). 

Medium 

— Brightness 
(readability  in  day 
light  &  dark  condi¬ 
tions) 

Screen  must  be  light  emitting. 

Screens  without  backlighting 
unacceptable. 

High 

—Resolution 

70  lines  per  inch  resolution  acceptable 
for  most  applications.  Complex  gra¬ 
phics  require  higher  resolution. 

Medium 

— Update  Rate 

Long  update  rate  (15  sec.  or  greater) 
clearly  unacceptable.  Human  factors 
guidelines  are  for  no  greater  than  2 
seconds. 

Medium 

— Function  Keys 

Limited  set  okay  for  field.  Full  keyboard 
desirable  in  shop. 

Medium 

Ecrtabjlity/ 

Ru, goodness 

— Weight 

About  27  pounds  too  heavy.  Fifteen 
pounds  or  less  acceptable.  Lighter  is 
better. 

High 
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Functional 

Characteristic 

PEAM 

Evaluation  Findings 

Degree  of  Support 
from  Evaluation 

—Size 

Key  to  size  is  technicians'  perception 
that  device  is  portable.  Size  of  small 
suitcase  or  attache  case  okay.  If  dis¬ 
play  is  detachable,  it  should  be  made 
easy  to  hang  up. 

Medium 

— Cables 

Minimum  use  of  cables,  desirable  (e.g., 
for  microphone,  power  connections  to 
weapons  systems).  Use  radio  or  Infra- 
Red  link  for  interactive  voice. 

High 

—Battery  Opera¬ 
tion 

4  hours  acceptable  for  most  field 
operations;  more  hours  desirable. 

Medium 

— Power  Supply 

12V  DC  from  rechargeable  battery  or 

AC  converter. 

Medium 

— Militarized 
(Ruggedness) 

Necessary  for  field  operation.  Not 
required  for  most  shop  uses.  Standard 
microcomputer  acceptable  for  many 
applications. 

Low 

— Interactive 

Voice 

Interactive  voice  in  improved  form  may 
be  desirable  for  certain  applications. 
Intelligibility  needs  improvement. 

Needs  further  study. 

V.cd'um 

— Authoring 
System 

•Development  of  authoring  system 
should  have  high  priority  for  follow-on 
work. 

High 

•Automatic  branching  helped  reduce 
error  rate  considerably  (6:1  for 
troubleshooting). 

High 

•Authoring  software  needs  to  be  sup¬ 
ported  by  standard  operating  system 
(e.g.,  MS  DOS,  UNIX). 

High 

•Easy  way  to  enter  graphics  needs 
development. 

High 
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Functional 

PEAM 

Degree  of  Support 

Characteristic 

Evaluation  Findings 

from  Evaluation 

•Authoring  workstation  must  include 
delivery  device. 

High 

•Data  compression  techniques  worked 
well;  should  be  considered  for  future 
applications  if  memory  is  a  restriction. 

High 

•Routines  to  check  branching,  com¬ 
mand  lines,  etc.,  a  requirement. 

High 

Technical  Infor- 
mation  Data 

— Organization/ 
Screen  Formats 

•Ease  of  moving  through  the  data  (for 
more  detail  or  to  different  tasks)  a 
mandatory  feature  as  demonstrated. 
Further  development  of  information  or¬ 
ganization  techniques  required. 

High 

•Different  levels  for  technician  experi¬ 
ence  level  is  necessary  for  most  appli¬ 
cations. 

High 

•Illustrated  job  aids  worked  well.  More 
development  needed  for  complex 
schematics  such  as  signal  flow 
diagrams. 

High 

—Training  Device 
vs.  Field  Job  AiC 

PEAM  as  a  job  aid  was  tested  in  Army 
application  and  worked  well.  Feasi¬ 
bility  of  combining  job  aiding  with 
training  should  be  further  considered. 

High 

21 


